1. Field of the Invention
The present invention relates to image restoration and more particularly to applying total variational regulation to a blind image.
2. Description of the Prior Art
Digital image restoration has received considerable attention in the field of image processing and computer vision during the past two decades. It contains applications in astronomial imaging, X-ray and MRI medical imaging, degraded media (film or videotape) restoration, artifact removal in image coding, law enforcement, etc. The problem of image restoration with known blur functions has been well studied in the past. Inverse filtering and regularized least squares are two of the most popular approaches for this problem. These are further described by M. R. Banham and A. K. Katsaggelos, in xe2x80x9cDigital Image Restorationxe2x80x9d, IEEE Signal Processing Magazine, Vol. 14, No. 2, pp. 24-41, 1997; and by A. K. Katsaggelos in Digital Image Restoration, Springer-Verlag, New York, 1991. Recently, researchers have focused on the more challenging problem, which is to recover the true image together with the blur function from the degraded image. This problem is called blind image restoration or blind image deconvolution. This problem is more practical since the blur function is usually unknown for most cases. D. Kundur and D. Hatzinakos in xe2x80x9cBlind Image deconvolutionxe2x80x9d, IEEE Signal Processing Magazine, Vol. 13, No.3, pp. 43-64, 1996, gave a review of previous methods, which include methods based on the frequency domain zeros in the PSF (point spread function) and the methods based on representing the blurred image as an ARMA (autoregressive moving average) process. The point spread function is further described by T. G. Stockham Jr., T. M. Cannon, and R. B. Ingebresten in xe2x80x9cBlind Deconvolution Through Digital Signal Processingxe2x80x9d, Proc. IEEE, Vol. 63, pp. 678-692, 1975; and by R. L. Lagendijk, A. M. Tekalp, and J. Biemond in xe2x80x9cMaximum Likelihood Image And Blur Identificationxe2x80x9d, Optical Engineering, Vol. 29, No.5, pp. 422-435, 1990. The autoregressive moving average is further described by A. M. Tekalp, H. Kaufman, and J. Woods in xe2x80x9cIdentification Of Image And Blur Parameters For The Restoration Of Noncausal Blursxe2x80x9d, IEEE Trans. Acoust., Speech, Signal Processing, Vol. 34, pp. 963-972, 1986; and by K. T. Lay and A. K. Katsaggelos in xe2x80x9cImage Identification And Restoration Based On Expectation-maximization Algorithmxe2x80x9d, Optical Engineering, Vol. 29, No. 5, pp. 436-445, 1990. The methods based on the frequency domain zeros are restricted to some specific types of blurs, such as motion blurs or defocused blurs with a circular aperture. In addition, these methods are sensitive to the additive noises. The methods based on the ARMA model are suspensible to some artifact in the restored image due to the artificial assumption that the true image is a two-dimensional autoregressive (AR) process.
More recently, Y.-L. You and M. Kaveh in xe2x80x9cA Regularization Approach To Joint Blur Identification And Image Restorationxe2x80x9d, IEEE Trans. Image Processing, Vol. 5, No. 3, pp. 416-428, 1996 developed a regularization approach for the blind image restoration problem, which is an ill-posed inverse problem due to the fact that the solution is not unique. Their regularization approach imposes additional smoothness constraint on the true image as well as the blur function, thereby making the problem well posed. This leads to an image-blur coupled function optimization problem. An alternating minimization algorithm with the conjugate gradient iterations was proposed to solve this coupled optimization problem.
You and Kaveh""s regularization formulation used an H1 norm for the smoothness constraint on the image and the blur. This smoothness constraint prohibits discontinuities in the solution. Unfortunately, there usually exist sharp discontinuities in the true images. The desirable details of the true image can be lost due to the smoothed discontinuities. Although You and Kaveh proposed a weighted regularization to alleviate this problem, this weighting scheme is somehow ad hoc and the parameters involved in this scheme may need to be tuned in a case by case basis.
A new regularization approach that employed the total variation (TV) norm in stead of the standard H1 norm for the image constraint was proposed by L. Rudin, S. Osher, and E. Fatemi in xe2x80x9cNonlinear Total Variation Based Noise Removal Algorithmsxe2x80x9d, Physica D., Vol. 60, pp. 259-268, 1992 for the image denoising problem. The TV regularization has been proved to be capable of preserving discontinuities while imposing smoothness constraints and it is effective for recovering blocky images. T. F. Chan and C. K. Wong in xe2x80x9cTotal Variation Blind Deconvolutionxe2x80x9d, IEEE Trans. Image Processing, Vol. 7, No. 3, pp. 370-375, 1998 modified You and Kaveh""s regularization formulation by using the TV norm for the smoothness constraint on the image as well as the blur function instead of the H1 norm, thus preserving the discontinuities in the recovered image function.
The present invention provides a new formulation for blind image restoration from an image sequence. Total variational (TV) regularization is employed to allow discontinuities in a true image function. An iterative alternating algorithm using quasi-Newton iterations is provided to solve an image-blur coupled nonlinear optimization problem. This formulation is then extended to the blind image restoration from an image sequence by introducing motion parameters into a multi-frame data constraint.
The input to the blind image restoration system of the present invention contains an image sequence and initial values for image blur and motion parameters. Within an image blur parameters updater, the system first updates the image blur parameters by using Quasi-Newton iterations to minimize the energy function with the motion parameters and restored image, fixed with their current values. After that, the signal flows to a motion parameters updater where the motion parameters between subsequent frames in the image sequence are updated by using Newton iterations to minimize the energy function with the blur parameters and restored image, fixed with their current values. The restored image is then updated in a restored image updater by using a preconditioned conjugate gradient algorithm to minimize the energy function derived from the total variational (TV) regularization formulation. The TV-based energy function is then computed in a TV-based energy function computer by using the currently updated parameter values. Within a converged decider, if the relative difference between the current energy function value and the energy value computed in the previous iteration is within a threshold, then it is converged and the restored image is outputted. If it has not converged, the signal flows back to the image blur parameters updater to update the parameters.